⚡ Real-Time Stock Insight & Prediction Platform- finance-lake

A scalable, end-to-end data engineering and machine learning platform that ingests real-time stock market data, processes it using Spark Structured Streaming & Delta Lake, and generates AI-driven price predictions using Spark MLlib (Random Forest). Results are visualized in a live Streamlit dashboard.

� Contributors

Developed by:

• Abdessamia KOUBRITI
• Abdelali AIT CHIKH
• Larbi ASMAOUI

🖼️ Dashboard Preview

(Main dashboard view showing real-time price and signals)

🎥 Demo Video

▶️ Watch the Project Demo

🚀 Features

• Real-Time Ingestion: Fetches live stock data (Yahoo Finance/AlphaVantage) and pushes to Redpanda (Kafka).
• Lakehouse Architecture: Implements a Medallion Architecture (Bronze used for raw, Silver for clean, Gold for aggregated) using Delta Lake on MinIO (S3).
• Spark Structured Streaming: Stateful processing, watermark handling, and windowed aggregations.
• Machine Learning:
  • Feature Engineering: Real-time feature computation (Volatility, Log Returns) using native Spark functions.
  • Model: Random Forest Classifier implemented with Spark MLlib for robust price direction prediction.
• Interactive Dashboard: Streamlit app powered by DuckDB for low-latency queries on Delta tables.
• Infrastructure as Code: Entire stack dockerized with Docker Compose.

🛠️ Tech Stack

• Ingestion: Python, Kafka Producer
• Message Broker: Redpanda (Kafka compatible)
• Processing: Apache Spark 3.4 (Structured Streaming)
• Storage: MinIO (S3 compatible object storage)
• Table Format: Delta Lake 2.4
• Machine Learning: Spark MLlib (Random Forest)
• Visualization: Streamlit, Plotly, DuckDB
• Containerization: Docker, Docker Compose

🏗️ Architecture Overview

The pipeline follows a modern "Medallion" Lakehouse architecture:

1. Source: kafka_producer fetches ticks and publishes to stock_prices topic.
2. Bronze Layer: Spark reads Kafka, dumps raw bytes/json to Delta table (s3a://finance-lake/lake/bronze).
3. Silver Layer: Deduplicates data, enforces schema, and cleans types (s3a://finance-lake/lake/silver).
4. Gold Layer: Aggregates data into time windows (e.g., 1-minute candles) and calculates financial metrics (s3a://finance-lake/lake/gold).
5. Machine Learning:
   • Training: Spark MLlib trains a Random Forest model on historical Gold data (src/ml/train_rf.py).
   • Inference: Real-time spark job consumes Gold data, predicts "BUY/SELL/HOLD", and writes to predictions table (src/ml/stock_inference.py).
6. Serving: Streamlit connects to MinIO via DuckDB to serve the dashboard.

⚙️ Setup & Installation

Prerequisites

• Docker & Docker Compose installed.
• (Optional) git installed.

Configure Environment

The project comes with reasonable defaults.

Start the Platform

Run the entire stack in detached mode:

docker compose up -d

Note: The first run may take a few minutes to build the Spark and Scraper images.

🖥️ Usage

Accessing Services

Triggering ML Training

To retrain the Random Forest model on the latest data:

docker compose exec spark-streaming spark-submit \
  --master "local[*]" \
  --driver-memory 4g \
  /app/src/ml/train_rf.py

Running Inference

To run the inference job manually:

docker compose exec spark-streaming spark-submit \
  --master "local[*]" \
  --driver-memory 4g \
  /app/src/ml/stock_inference.py

Checking Logs

To monitor the streaming process:

docker compose logs -f spark-streaming

📂 Project Structure

├── docker-compose.yml       # Infrastructure definition
├── Dockerfile.spark         # Custom Spark image with Jars
├── Dockerfile.dashboard     # Streamlit app image
├── src/
│   ├── ingestion/           # Kafka Producers
│   ├── processing/          # Spark Structured Streaming Jobs
│   ├── ml/                  # Spark MLlib Random Forest (Train & Inference)
│   ├── dashboard/           # Streamlit App
│   └── debug_*.py           # Utilities for inspecting Delta tables
├── jars/                    # Spark dependencies (Delta, Kafka, Hadoop-AWS)
├── data/                    # Local mount for checkpoints
└── logs/                    # Application logs

📸 Additional Screenshots

MinIO Data Lake Buckets

(Bronze, Silver, and Gold data layers in MinIO)